Textual Inversion for medical image generation
Popular for text-to-image modeling
Stable Diffusion: open source + inference on single GPU
🚨 Training requires a lot of compute and data
🏥 Medical domain can have rare diseases and local compute
🧐 Does Stable Diffusion already know medical imaging?
✒️ “a prostate MRI scan”
✒️ “a T2-weigthed MRI scan of a prostate”
💡 Fine-tune model
Methods fine-tune sub-parts of diffusion model
✒️ Textual Inversion only trains token embedding
Original work uses small embeddings and ~5 examples
💡 Use larger embeddings and more examples
🚧 image visualizing setup
| #Real | #Synthetic | AUC - Prostate MRI |
|---|---|---|
| 200 | 0 | 0.780 ± 0.017 |
| 200 | 2000 | 0.803 ± 0.009 |
📈 Adding synthetic cases maintains or improves performance
| #Real | #Synthetic | AUC - Prostate MRI |
|---|---|---|
| 200 | 0 | 0.780 ± 0.017 |
| 200 | 2000 | 0.803 ± 0.009 |
| 0 | 2000 | 0.766 ± 0.020 |
📈 Adding synthetic cases maintains or improves performance
📉 Using only synthetic data gives small performance drop
| #Real | #Synthetic | AUC - Prostate MRI |
|---|---|---|
| 200 | 0 | 0.780 ± 0.017 |
| 200 | 2000 | 0.803 ± 0.009 |
| 0 | 2000 | 0.766 ± 0.020 |
| 0 | 2000* | 0.562 ± 0.036 |
📈 Adding synthetic cases maintains or improves performance
📉 Using only synthetic data gives small performance drop
👎 Training on 10-case embeddings shows quality difference
🧐 What about GANs?
🔬 Fine-tune a pre-trained StyleGAN3 on 100 images
⚖️ Similar training time and compute
🧑⚕️ Prostate radiologist preferred diffusion model (36/50)
↔︎️ Interpolate between healthy and diseased state
⚡ Train two embeddings: healthy, diseased
✒️ “healthy:30% AND diseased:70%”
➕ Combine multiple embeddings to show multiple diseases
⚡ Train embeddings per disease
✒️ “pleural_effusion AND pneumonia”
IMAGE (full prostate image in a couple fragments)
🚧 style this page more
Scan for paper, code, contact details & more!